Question of the Week: Are Randomized Controlled Trials a Good Way to Evaluate Development Projects?

Question of the Week posts review important questions and controversies in global development by providing background information and links to a full spectrum of analysis and opinion. Today’s post tackles the debate over randomized controlled trials (RCTs), which have emerged as an important tool for measuring the success of development interventions.

International donors have spent well over $2 trillion in development assistance over the past five decades, but there remains significant uncertainty about what works and doesn’t work to reduce poverty and grow economies. Some scholars even argue that development aid makes life worse, not better, for poor people in the global south. And in an increasingly tight fiscal environment, governments and the public understandably want accountability to ensure that scarce dollars are making measurable impacts. In this context, randomized controlled trials (RCTs)—a statistical tool adapted from biological and medical research—have become a prominent though controversial tool for evaluating aid effectiveness.

How does a randomized controlled trial work? In a RCT, a test group receives an intervention (say, a free insecticide-treated bed net or a new way of receiving payments by mobile phone) while a control group does not. Members of each group are randomly selected so as to make both groups statistically identical and representative of the larger population. In theory, then, the only difference between the two is whether or not they receive the intervention. Outcomes of interest (such as participants’ income or health status) can then be compared to evaluate the intervention’s impact.

Supporters believe that RCTs can provide precise and rigorous findings about what works in development—findings unavailable through other types of evaluations. This can prevent governments and non-profits from spending aid dollars on ineffective interventions. "The experimental approach is something different and potentially transformative: it not only produces relatively clear-cut evidence that is hard to ignore, but also forces economists to engage development problems where they play out,” write Rachel Glennerster and Michael Kremer of Harvard University and J-PAL, an MIT-based network of researchers who use RCTs to evaluate international development.

RCTs have also attracted substantial criticism. One challenge, known as the “external validity problem,” is whether findings from a trial in one context will be true elsewhere. Different countries, of course, have different political environments, economic systems, poverty levels, and more—and these factors can all contribute to the success or failure of antipoverty interventions. As NYU economist Bill Easterly points out, “If you find that using flipcharts in classrooms raises test scores in one experiment, does that mean that aid agencies should buy flipcharts for every school in the world? Context matters–the effect of flipcharts depends on the existing educational level of students and teachers, availability of other educational methods, and about a thousand other things.” RCT results can be generalized in medicine because all humans are biologically comparable, some argue, but comparisons are more problematic in the realm of human behavior.

Responding to the external validity problem, proponents of RCTs argue that the collective results of these trials provide invaluable knowledge about how humans behave and which antipoverty interventions work. “…we have accumulated enough knowledge in some domains (primary education, for example), that we are beginning to have a broad systemic view of problems and solutions…” explains MIT economist Esther Duflo, a co-founder of J-PAL.

More damningly, some critics also argue that RCTs are simply unable to answer many of the biggest questions in the foreign assistance debate. “What would be the effects of disbursing $1-1.5 billion of foreign aid to Pakistan?” asks Arvind Subramanian, a senior fellow at the Center for Global Development. “RCTs do not, and cannot, have anything to say on the matter—not only because of their narrow focus and applicability, and hence non-generalizability, but also because they cannot speak to macroeconomic effects.” Likewise, it would be nearly impossible to randomly test the impact of major development investments like roads, ports, and telecommunications infrastructure.

Indeed, even the most dedicated supporters of RCTs do not argue that these trials can answer every question. “…a common misperception directed at advocates of RCTs is that we suggest that they can and should be conducted on every program…” writes Dean Karlan, professor of economics at Yale and founder of Innovations for Poverty Action, a non-profit focused on evaluating development projects through RCTs.

Another major question surrounding RCTs is ethics. Ethical guidelines for medical RCTs involving human subjects are often well-developed, with participants aware that they are participating in an experiment and that they may receive either a medicine or a placebo. Test subjects in antipoverty RCTs, by contrast, might not be told of their participation in an experiment. Critics point to this lack of informed consent as an important issue. Additionally, some argue is that it is unethical for researchers to withhold potentially lifesaving interventions from randomly selected subjects when they could distribute these interventions more broadly. “If we don’t know who needs the treatment and what it will do—as long as it does no harm—then deliberately withholding that treatment for the purpose of an experiment seems ethically harmless. But the randomistas surely exaggerate our ignorance about the efficacy of the things we do in the name of fighting poverty. We know that de-worming tablets (say) work almost always,” says Martin Ravallion, acting chief economist of the World Bank.

At their best, RCTs can provide important information about many kinds of programs that can improve the lives of the world’s poorest people. However, as such scholars as Harvard professor Dani Rodrik have pointed out, RCTs are not a “one size fits all” evaluative tool. In the future, governments, international organizations, and non-profits could conceivably require RCTs for their projects, and funding could come to depend on RCT results. While this can help ensure efficacy and accountability in many cases, it is important to ensure that important investments in areas unsuitable for RCTs are not ignored.

What do you think about the benefits and drawbacks of randomized controlled trials? What are the best ways to measure the success of development projects? Let us know in the comments below, and stay tuned for new Question of the Week posts soon.